Sparse PCA via Covariance Thresholding

In sparse principal component analysis we are given noisy observations of a low-rank matrix of dimension n × p and seek to reconstruct it under additional sparsity assumptions. In particular, we assume here each of the principal components v1, . . . ,vr has at most s0 non-zero entries. We are particularly interested in the high dimensional regime wherein p is comparable to, or even much larger than n. In an influential paper, Johnstone and Lu (2004) introduced a simple algorithm that estimates the support of the principal vectors v1, . . . ,vr by the largest entries in the diagonal of the empirical covariance. This method can be shown to identify the correct support with high probability if s0 ≤ K1 √ n/ log p, and to fail with high probability if s0 ≥ K2 √ n/ log p for two constants 0 < K1,K2 < ∞. Despite a considerable amount of work over the last ten years, no practical algorithm exists with provably better support recovery guarantees. Here we analyze a covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler, Vilenchik, et al. (2015). On the basis of numerical simulations (for the rank-one case), these authors conjectured that covariance thresholding correctly recover the support with high probability for s0 ≤ K √ n (assuming n of the same order as p). We prove this conjecture, and in fact establish a more general guarantee including higher-rank as well as n much smaller than p. Recent lower bounds (Berthet and Rigollet, 2013; Ma and Wigderson, 2015) suggest that no polynomial time algorithm can do significantly better. The key technical component of our analysis develops new bounds on the norm of kernel random matrices, in regimes that were not considered before. Using these, we also derive sharp bounds for estimating the population covariance, and the principal component (with `2-loss). c ©2016 Yash Deshpande and Andrea Montanari. Deshpande and Montanari